20 research outputs found
CHAD: Combinatory Homomorphic Automatic Differentiation
We introduce Combinatory Homomorphic Automatic Differentiation (CHAD), a
principled, pure, provably correct define-then-run method for performing
forward and reverse mode automatic differentiation (AD) on programming
languages with expressive features. It implements AD as a compositional,
type-respecting source-code transformation that generates purely functional
code. This code transformation is principled in the sense that it is the unique
homomorphic (structure preserving) extension to expressive languages of
Elliott's well-known and unambiguous definitions of AD for a first-order
functional language. Correctness of the method follows by a (compositional)
logical relations argument that shows that the semantics of the syntactic
derivative is the usual calculus derivative of the semantics of the original
program.
In their most elegant formulation, the transformations generate code with
linear types. However, the code transformations can be implemented in a
standard functional language lacking linear types: while the correctness proof
requires tracking of linearity, the actual transformations do not. In fact,
even in a standard functional language, we can get all of the type-safety that
linear types give us: we can implement all linear types used to type the
transformations as abstract types, by using a basic module system.
In this paper, we detail the method when applied to a simple higher-order
language for manipulating statically sized arrays. However, we explain how the
methodology applies, more generally, to functional languages with other
expressive features. Finally, we discuss how the scope of CHAD extends beyond
applications in AD to other dynamic program analyses that accumulate data in a
commutative monoid.Comment: arXiv admin note: substantial text overlap with arXiv:2007.0528
Efficient CHAD
We show how the basic Combinatory Homomorphic Automatic Differentiation
(CHAD) algorithm can be optimised, using well-known methods, to yield a simple
and generally applicable reverse-mode automatic differentiation (AD) technique
that has the correct computational complexity that we would expect of a reverse
AD algorithm. Specifically, we show that the standard optimisations of sparse
vectors and state-passing style code (as well as defunctionalisation/closure
conversion, for higher-order languages) give us a purely functional algorithm
that is most of the way to the correct complexity, with (functional) mutable
updates taking care of the final log-factors. We provide an Agda formalisation
of our complexity proof. Finally, we discuss how the techniques apply to
differentiating parallel functional programs: the key observations are 1) that
all required mutability is (commutative, associative) accumulation, which lets
us preserve task-parallelism and 2) that we can write down data-parallel
derivatives for most data-parallel array primitives
Efficient Dual-Numbers Reverse AD via Well-Known Program Transformations
Where dual-numbers forward-mode automatic differentiation (AD) pairs each scalar value with its tangent value, dual-numbers \emph{reverse-mode} AD attempts to achieve reverse AD using a similarly simple idea: by pairing each scalar value with a backpropagator function. Its correctness and efficiency on higher-order input languages have been analysed by Brunel, Mazza and Pagani, but this analysis used a custom operational semantics for which it is unclear whether it can be implemented efficiently. We take inspiration from their use of \emph{linear factoring} to optimise dual-numbers reverse-mode AD to an algorithm that has the correct complexity and enjoys an efficient implementation in a standard functional language with support for mutable arrays, such as Haskell. Aside from the linear factoring ingredient, our optimisation steps consist of well-known ideas from the functional programming community. We demonstrate the practical use of our technique by providing a performant implementation that differentiates most of Haskell98
Dual-Numbers Reverse AD, Efficiently
Where dual-numbers forward-mode automatic differentiation (AD) pairs each scalar value with its tangent derivative, dual-numbers /reverse-mode/ AD attempts to achieve reverse AD using a similarly simple idea: by pairing each scalar value with a backpropagator function. Its correctness and efficiency on higher-order input languages have been analysed by Brunel, Mazza and Pagani, but this analysis was on a custom operational semantics for which it is unclear whether it can be implemented efficiently. We take inspiration from their use of /linear factoring/ to optimise dual-numbers reverse-mode AD to an algorithm that has the correct complexity and enjoys an efficient implementation in a standard functional language with resource-linear types, such as Haskell. Aside from the linear factoring ingredient, our optimisation steps consist of well-known ideas from the functional programming community. Furthermore, we observe a connection with classical imperative taping-based reverse AD, as well as Kmett's 'ad' Haskell library, recently analysed by Krawiec et al. We demonstrate the practical use of our technique by providing a performant implementation that differentiates most of Haskell98
Subcutaneous or Transvenous Defibrillator Therapy.
BACKGROUND: The subcutaneous implantable cardioverter-defibrillator (ICD) was designed to avoid complications related to the transvenous ICD lead by using an entirely extrathoracic placement. Evidence comparing these systems has been based primarily on observational studies. METHODS: We conducted a noninferiority trial in which patients with an indication for an ICD but no indication for pacing were assigned to receive a subcutaneous ICD or transvenous ICD. The primary end point was the composite of device-related complications and inappropriate shocks; the noninferiority margin for the upper boundary of the 95% confidence interval for the hazard ratio (subcutaneous ICD vs. transvenous ICD) was 1.45. A superiority analysis was prespecified if noninferiority was established. Secondary end points included death and appropriate shocks. RESULTS: A total of 849 patients (426 in the subcutaneous ICD group and 423 in the transvenous ICD group) were included in the analyses. At a median follow-up of 49.1 months, a primary end-point event occurred in 68 patients in the subcutaneous ICD group and in 68 patients in the transvenous ICD group (48-month Kaplan-Meier estimated cumulative incidence, 15.1% and 15.7%, respectively; hazard ratio, 0.99; 95% confidence interval [CI], 0.71 to 1.39; P = 0.01 for noninferiority; P = 0.95 for superiority). Device-related complications occurred in 31 patients in the subcutaneous ICD group and in 44 in the transvenous ICD group (hazard ratio, 0.69; 95% CI, 0.44 to 1.09); inappropriate shocks occurred in 41 and 29 patients, respectively (hazard ratio, 1.43; 95% CI, 0.89 to 2.30). Death occurred in 83 patients in the subcutaneous ICD group and in 68 in the transvenous ICD group (hazard ratio, 1.23; 95% CI, 0.89 to 1.70); appropriate shocks occurred in 83 and 57 patients, respectively (hazard ratio, 1.52; 95% CI, 1.08 to 2.12). CONCLUSIONS: In patients with an indication for an ICD but no indication for pacing, the subcutaneous ICD was noninferior to the transvenous ICD with respect to device-related complications and inappropriate shocks. (Funded by Boston Scientific; PRAETORIAN ClinicalTrials.gov number, NCT01296022.)
CHAD: Combinatory Homomorphic Automatic Differentiation
We introduce Combinatory Homomorphic Automatic Differentiation (CHAD), a principled, pure, provably correct define-then-run method for performing forward and reverse mode automatic differentiation (AD) on programming languages with expressive features. It implements AD as a compositional, type-respecting source-code transformation that generates purely functional code. This code transformation is principled in the sense that it is the unique homomorphic (structure preserving) extension to expressive languages of Elliott’s well-known and unambiguous definitions of AD for a first-order functional language. Correctness of the method follows by a (compositional) logical relations argument that shows that the semantics of the syntactic derivative is the usual calculus derivative of the semantics of the original program. In their most elegant formulation, the transformations generate code with linear types. However, the code transformations can be implemented in a standard functional language lacking linear types: While the correctness proof requires tracking of linearity, the actual transformations do not. In fact, even in a standard functional language, we can get all of the type-safety that linear types give us: We can implement all linear types used to type the transformations as abstract types by using a basic module system. In this article, we detail the method when applied to a simple higher-order language for manipulating statically sized arrays. However, we explain how the methodology applies, more generally, to functional languages with other expressive features. Finally, we discuss how the scope of CHAD extends beyond applications in AD to other dynamic program analyses that accumulate data in a commutative monoid
CHAD: Combinatory Homomorphic Automatic Differentiation
We introduce Combinatory Homomorphic Automatic Differentiation (CHAD), a principled, pure, provably correct define-then-run method for performing forward and reverse mode automatic differentiation (AD) on programming languages with expressive features. It implements AD as a compositional, type-respecting source-code transformation that generates purely functional code. This code transformation is principled in the sense that it is the unique homomorphic (structure preserving) extension to expressive languages of Elliott's well-known and unambiguous definitions of AD for a first-order functional language. Correctness of the method follows by a (compositional) logical relations argument that shows that the semantics of the syntactic derivative is the usual calculus derivative of the semantics of the original program. In their most elegant formulation, the transformations generate code with linear types. However, the code transformations can be implemented in a standard functional language lacking linear types: while the correctness proof requires tracking of linearity, the actual transformations do not. In fact, even in a standard functional language, we can get all of the type-safety that linear types give us: we can implement all linear types used to type the transformations as abstract types, by using a basic module system. In this paper, we detail the method when applied to a simple higher-order language for manipulating statically sized arrays. However, we explain how the methodology applies, more generally, to functional languages with other expressive features. Finally, we discuss how the scope of CHAD extends beyond applications in AD to other dynamic program analyses that accumulate data in a commutative monoid
Efficient Dual-Numbers Reverse AD via Well-Known Program Transformations
Where dual-numbers forward-mode automatic differentiation (AD) pairs each
scalar value with its tangent value, dual-numbers \emph{reverse-mode} AD
attempts to achieve reverse AD using a similarly simple idea: by pairing each
scalar value with a backpropagator function. Its correctness and efficiency on
higher-order input languages have been analysed by Brunel, Mazza and Pagani,
but this analysis used a custom operational semantics for which it is unclear
whether it can be implemented efficiently. We take inspiration from their use
of \emph{linear factoring} to optimise dual-numbers reverse-mode AD to an
algorithm that has the correct complexity and enjoys an efficient
implementation in a standard functional language with support for mutable
arrays, such as Haskell. Aside from the linear factoring ingredient, our
optimisation steps consist of well-known ideas from the functional programming
community.
We demonstrate the practical use of our technique by providing a performant
implementation that differentiates most of Haskell98.Comment: Short version of arXiv:2205.1136
CHAD: Combinatory Homomorphic Automatic Differentiation
We introduce Combinatory Homomorphic Automatic Differentiation (CHAD), a principled, pure, provably correct define-then-run method for performing forward and reverse mode automatic differentiation (AD) on programming languages with expressive features. It implements AD as a compositional, type-respecting source-code transformation that generates purely functional code. This code transformation is principled in the sense that it is the unique homomorphic (structure preserving) extension to expressive languages of Elliott's well-known and unambiguous definitions of AD for a first-order functional language. Correctness of the method follows by a (compositional) logical relations argument that shows that the semantics of the syntactic derivative is the usual calculus derivative of the semantics of the original program. In their most elegant formulation, the transformations generate code with linear types. However, the code transformations can be implemented in a standard functional language lacking linear types: while the correctness proof requires tracking of linearity, the actual transformations do not. In fact, even in a standard functional language, we can get all of the type-safety that linear types give us: we can implement all linear types used to type the transformations as abstract types, by using a basic module system. In this paper, we detail the method when applied to a simple higher-order language for manipulating statically sized arrays. However, we explain how the methodology applies, more generally, to functional languages with other expressive features. Finally, we discuss how the scope of CHAD extends beyond applications in AD to other dynamic program analyses that accumulate data in a commutative monoid
Dual-Numbers Reverse AD, Efficiently
Where dual-numbers forward-mode automatic differentiation (AD) pairs each scalar value with its tangent derivative, dual-numbers /reverse-mode/ AD attempts to achieve reverse AD using a similarly simple idea: by pairing each scalar value with a backpropagator function. Its correctness and efficiency on higher-order input languages have been analysed by Brunel, Mazza and Pagani, but this analysis was on a custom operational semantics for which it is unclear whether it can be implemented efficiently. We take inspiration from their use of /linear factoring/ to optimise dual-numbers reverse-mode AD to an algorithm that has the correct complexity and enjoys an efficient implementation in a standard functional language with resource-linear types, such as Haskell. Aside from the linear factoring ingredient, our optimisation steps consist of well-known ideas from the functional programming community. Furthermore, we observe a connection with classical imperative taping-based reverse AD, as well as Kmett's 'ad' Haskell library, recently analysed by Krawiec et al. We demonstrate the practical use of our technique by providing a performant implementation that differentiates most of Haskell98